Papers with speech-to-text translation
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020.aacl-demo)
Copied to clipboard
| Challenge: | End-to-end sequence-to sequence (S2S) modeling has witnessed rapid growth in speech-totext (ST) tasks. |
| Approach: | They introduce fairseq S2T, a fairsq extension for speech-to-text modeling tasks such as end-to end speech recognition and speech-text translation. |
| Outcome: | The proposed extension provides end-to-end workflows from data pre-processing, model training to offline (online) inference. |
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for speech-to-text translation (ST) have achieved impressive supervised and zero-shot performance. |
| Approach: | They propose to use consistency regularization methods to boost end-to-end (E2E) speech-totext translation (ST) by regularizing the intra-modal consistency instead of the modality gap. |
| Outcome: | The proposed training strategies achieve state-of-the-art (SOTA) performance in most translation directions. |
LibriS2S: A German-English Speech-to-Speech Translation Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in speech-to-text translation have led to significant improvements, but the availability of appropriate training data is limiting. |
| Approach: | They propose a new text-to-speech and speech-tospech translation model that directly learns to generate the speech signal based on the pronunciation of the source language. |
| Outcome: | The proposed model learns to generate speech signal based on pronunciation of source language. |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects (2024.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Adelani, Daud Abolade, Noah Smith, Yulia Tsvetkov
| Challenge: | Recent efforts to develop NLP tools for low-resource languages focus on their standard dialects. |
| Approach: | They propose a high-quality parallel text and speech corpus for Yoruba . they use native speakers to collect data from four regional yoruba dialects . |
| Outcome: | The proposed dataset shows that dialect-adaptive finetuning can narrow performance disparities . the dataset will be released publicly under an open license . |
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect. |
| Approach: | They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model. |
| Outcome: | The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data. |
DUB: Discrete Unit Back-translation for Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Discrete unit back-translation (DUB) is a back-translated speech-to-text translation (ST) technique that can be applied to ST . a modality gap between speech and text makes it difficult to transfer these techniques to ST due to the modality of the speech-text model. |
| Approach: | They propose a method to represent speech with discrete units instead of continuous features in direct ST. |
| Outcome: | The proposed method achieves comparable performance to existing methods that rely on large-scale external data. |
End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation (2023.emnlp-main)
Copied to clipboard
Juan Pablo Zuluaga-Gomez, Zhaocheng Huang, Xing Niu, Rohit Paturi, Sundararajan Srinivasan, Prashant Mathur, Brian Thompson, Marcello Federico
| Challenge: | Conventional speech-to-text translation systems are trained on single-speaker utterances, but they may not be applicable to real-life scenarios where the audio contains conversations by multiple speakers. |
| Approach: | They propose a speaker-turn-aware conversational speech translation model that integrates automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. |
| Outcome: | The proposed model outperforms the reference systems on the multi-speaker condition while attaining comparable performance on the single-speakspeaker conditions. |
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
UniCoM: A Universal Code-Switching Speech Generator (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code-switching (CS) is a common phenomenon in real-world conversations and poses significant challenges for multilingual speech technology. |
| Approach: | They propose a pipeline for generating high-quality, natural CS samples without altering sentence semantics. |
| Outcome: | The proposed pipeline generates high-quality, natural CS samples without altering sentence semantics without alteration of sentence semantic. |
Improving Language and Modality Transfer in Translation by Character-level Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Current translation systems cover only 5% of the world's languages . expanding to the long-tail of low-resource languages requires data-efficient methods that rely on cross-lingual and cross-modal knowledge transfer. |
| Approach: | They propose a character-based approach to improve adaptability to new languages and modalities by using a teacher-student approach and parallel translation data to obtain a SONAR character-level encoder. |
| Outcome: | The proposed model outperforms subword-based models in speech-to-text translation on the FLEURS benchmark on 33 languages and achieves state-of-the-art generalizability to unseen languages. |
It’s Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems (2025.acl-long)
Copied to clipboard
| Challenge: | idioms are defined as words with a figurative meaning not deducible from their individual components. |
| Approach: | They compare idiom translation as compared to conventional news translation in two languages . they compare MT and SLT systems with MT, Large Language Models and cascaded alternatives . |
| Outcome: | The proposed systems show better handling of idioms than standard news translation systems. |